Method and system for providing single view video signal based on a multiview video coding (MVC) signal stream

ABSTRACT

A method and system for producing a single view video signal based on a multiview video coding (MVC) signal stream. A MVC signal stream representing multiple spatially related views of a scene, including a base view and at least one dependent view, is decoded to provide multiple decoded video signals representing the spatially related views, with respective portions of the MVC signal stream representing one of multiple temporally adjacent video frames, and the MVC signal stream representing multiple sequences of spatially adjacent video frames. The decoded video signals are processed to provide a processed video signal representing one of the spatially related views using image information from more than one of the decoded video signals. As a result, more image data is used during processing, thereby improving the spatial and temporal image quality.

FIELD OF THE DISCLOSURE

The present disclosure relates to processing of multiview video coding(MVC) signals, and in particular, to processing of MVC signals for usein single-view displays.

BACKGROUND OF THE DISCLOSURE

With recent successes of three-dimensional (3D) feature length films, 3Dvideo services are expected to soar in popularity in the entertainmentand communication industries, and further, in the consumer electronicsindustry as more devices are designed to capture, produce, communicate,display and store such image information. For example, one applicationis that of conventional television using displays capable of rendering3D imaging, e.g., as a stereoscopic image, where the display renders twoviews, one for each eye, so that the images are perceived as 3D. Evenmore sophisticated displays will be able to produce and display multipleviews in such a way that the 3D image perceived by the viewer dependsupon the location of the viewer with respect to the display device. As aresult, a widely used video coding standard, identified as the H.264/AVCstandard, has been extended to include multiview video coding (MVC).

Referring to FIG. 1, multiview video sequences are captured by multipleimage capturing devices 12, 14, 16 positioned in different locations andcapturing images of the subjects A, B from different angles, e.g., aseach subject moves within the 3D space (along axes X, Y and Z), eachwith its own field of view 12 a, 14 a, 16 a, to produce correspondingsingle view video signals 13, 15, 17. (As will be readily appreciated,each of these video signals 13, 15, 17 can include multiple signals. Forexample, each may include multiple color specific signals, such as red,green and blue, or as is more common, a luminance component signal Y, ablue difference chrominance component signal Cb and a red differencechroma component signal Cr.) Accordingly, the captured images aredifferent representations of the same objects A, B. As a result, thereis some redundancy with respect to some of the captured imageinformation, but also complementary image information due to thedifferent viewing angles.

Referring to FIG. 2, the multiview video signals 13, 15, 17 are encodedby a MVC encoder 18 to produce a MVC signal stream 20. Such encoders arewell known in the art and can be implemented using various combinationsof hardware (e.g., one or more microprocessors or other suitableprocessors), firmware and software.

Referring to FIG. 3, in accordance with the MVC signal standard, theimaging capturing devices (e.g., digital video cameras) 12, 14, 16produce their signals 13, 15, 17 containing bitstreams 20, portions 22of which represent temporally adjacent video frames, and, as between therespective signals 13, 15, 17 from the image capturing devices, 12, 14,16, represent spatially adjacent video frames.

However, notwithstanding the anticipated popularity of 3D videoapplications, the fact is that many single view, e.g., two-dimensional(2D) imaging applications and display devices remain in use and areexpected to continue in use for some time. Currently, the MVC standardprovides backward compatibility such that a compliant decoder decodes asingle view and discards the remaining data, while a compliant MVCdecoder decodes all views to generate the multiview, e.g., 3D videoimage. It would be desirable to enhance the compatibility with singleview image devices.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a conceptual view of a conventional technique for capturingmultiview video sequences.

FIG. 2 depicts conventional encoding of multiview video signals to a MVCsignal stream.

FIG. 3 depicts an exemplary bitstream generated by a digital videoimaging capturing device.

FIG. 4 is a functional block diagram of a system for providing a singleview video signal based on a MVC signal stream in accordance with anexemplary embodiment.

FIG. 5 is a functional block diagram of an exemplary embodiment of thevideo processor of FIG. 4.

FIG. 6 depicts the use of temporally and spatially adjacent imageinformation from dependent views for processing the base view inaccordance with an exemplary embodiment.

FIG. 7 is a flowchart of an exemplary embodiment of a method ofproviding a single view video signal based on a MVC signal stream.

DETAILED DESCRIPTION OF PREFERRED EMBODIMENTS

The following detailed description is of example embodiments withreferences to the accompanying drawings. Such description is intended tobe illustrative and not limiting with respect to the scope of thepossible embodiments. Such embodiments are described in sufficientdetail to enable one of ordinary skill in the art to practice them, andit will be understood that other embodiments may be practiced with somevariations.

Throughout the present disclosure, absent a clear indication to thecontrary from the context, it will be understood that individual circuitelements as described may be singular or plural in number. For example,the terms “circuit” and “circuitry” may include either a singlecomponent or a plurality of components, which are either active and/orpassive and are connected or otherwise coupled together (e.g., as one ormore integrated circuit chips) to provide the described function.Additionally, the term “signal” may refer to one or more currents, oneor more voltages, or a data signal. Within the drawings, like or relatedelements will have like or related alpha, numeric or alphanumericdesignators. Further, while the present invention has been discussed inthe context of implementations using discrete electronic circuitry(preferably in the form of one or more integrated circuit chips), thefunctions of any part of such circuitry may alternatively be implementedusing one or more appropriately programmed processors, depending uponthe signal frequencies or data rates to be processed. Moreover, to theextent that the figures illustrate diagrams of the functional blocks ofvarious embodiments, the functional blocks are not necessarilyindicative of the division between hardware circuitry. Thus, forexample, one or more of the functional blocks (e.g., processors,memories, etc.) may be implemented in a single piece of hardware (e.g.,a general purpose signal processor, random access memory, hard diskdrive, etc.). Similarly, any programs described may be standaloneprograms, may be incorporated as subroutines in an operating system, maybe functions in an installed software package, etc.

Advantageously, embodiments of the invention, which include but are notlimited to a method and system, allow for more complete use of the imageinformation available from each of the multiple views in a MVC signalstream. By using more of the temporal or spatial, or both, imageinformation (than is available in a single signal stream) for a singleview image, improved image resolution and smoothing, particularly duringimage movement, can be achieved for the resulting single view image.Additionally, improved transcode quality can be achieved when convertingfrom a MVC signal to a single view signal with a codec.

In accordance with exemplary embodiments, a method and system areprovided for producing a single view video signal based on a multiviewvideo coding (MVC) signal stream. A MVC signal stream representingmultiple spatially related views of a scene, including a base view andat least one dependent view, is decoded to provide multiple decodedvideo signals representing the spatially related views, with respectiveportions of the MVC signal stream representing one of multipletemporally adjacent video frames, and the MVC signal stream representingmultiple sequences of spatially adjacent video frames. The decoded videosignals are processed to provide a processed video signal representingone of the spatially related views using image information from morethan one of the decoded video signals. As a result, more image data isused during processing, thereby improving the spatial and temporal imagequality.

In accordance with an exemplary embodiment, a method of providing asingle view video signal based on a multiview video coding (MVC) signalstream includes:

decoding a MVC signal stream representing a plurality of spatiallyrelated views of a scene, including a base view and at least onedependent view, to provide a plurality of decoded video signalsrepresenting the plurality of spatially related views; and

processing the plurality of decoded video signals using imageinformation from more than one of the plurality of decoded video signalsto provide a processed video signal representing one of the plurality ofspatially related views.

In accordance with another exemplary embodiment, a video system forproviding a single view video signal based on a multiview video coding(MVC) signal stream includes:

a decoder adapted to decode a MVC signal stream representing a pluralityof spatially related views of a scene, including a base view and atleast one dependent view, to provide a plurality of decoded videosignals representing the plurality of spatially related views; and

one or more video processors adapted to process the plurality of decodedvideo signals using image information from more than one of theplurality of decoded video signals to provide a processed video signalrepresenting one of the plurality of spatially related views.

In accordance with another exemplary embodiment, a computer readablemedium including a plurality of executable instructions that, whenexecuted by an integrated circuit design system, cause the integratedcircuit design system to produce at least one integrated circuit (IC)for providing a single view video signal based on a multiview videocoding (MVC) signal stream and including:

a decoder adapted to decode a MVC signal stream representing a pluralityof spatially related views of a scene, including a base view and atleast one dependent view, to provide a plurality of decoded videosignals representing the plurality of spatially related views; and

one or more video processors adapted to process the plurality of decodedvideo signals using image information from more than one of theplurality of decoded video signals to provide a processed video signalrepresenting one of the plurality of spatially related views.

Referring to FIG. 4, a system 100 in accordance with one exemplaryembodiment includes a MVC decoder 102 and a video processor 106,interconnected substantially as shown. The MVC decoder 102 is well knownin the art, e.g., implemented in accordance with the H.264/AVC standard,and can be implemented using various combinations of processing hardware(e.g., one or more microprocessors or other suitable processors),firmware and software. The incoming MVC signal stream 120 is decoded toproduce a signal 103 a representing the base view, plus additionalsignals 103 b, . . . , 103 n representing respective dependent views.The base view 104 a includes a sequence of tenporally adjacent viewframes 105 a (or fields). Similarly, the dependent views 104 b, . . . ,104 n include respective sequences of tenporally adjacent video frames105 b, . . . , 105 n. Further, as will be readily appreciated, due tothe location of the image capturing devices, e.g., similar to thepositioning of the image capture devices 12, 14, 16 in FIG. 1, the base104 a and dependent 104 b, . . . , 104 n views are mutually spatiallyadjacent. For example, using the arrangement of FIG. 1, if the centralimage capturing device 12 captures the base view 104 a, an adjacentimage capturing device 14 captures a dependent view 104 b which isspatially adjacent the base view 104 a captured by the center imagecapturing device 12, i.e., due to their overlapping fields of view 12 a,14 a.

These decoded signals 103 a, 103 b, . . . , 103 n are processed by thevideo processor 106 (discussed in more detail below) to produce a videosignal 107 representing a single view 108 containing tenporally adjacentvideo frames 109. Each of these frames 109 is based primarily on thecorresponding video frame 105 a of the base view 104 a, but alsoincludes image information from one or more temporal or spatial, orboth, video frames 105 b, . . . , 105 n from one or more neighboringdependent views 104 b, . . . , 104 n.

Referring to FIG. 5, an exemplary embodiment 106 a of the videoprocessor 106 of FIG. 4 includes a base view processor 120 a, dependentview processors 120 b, . . . , 120 n, and a final view processor 124,interconnected substantially as shown. As discussed in more detailbelow, the base view processor 120 a processes the base view signal 103a while the dependent view processors 120 b, . . . , 120 n process theirrespective dependent view signals 103 b, . . . , 103 n. Such processors120 a, 120 b, . . . , 120 n can be implemented using variouscombinations of hardware (e.g., one or more microprocessors or othersuitable processors), firmware and software in accordance with wellknown techniques. Each processor 120 a, 120 b, . . . , 120 n analyzesthe image data in its respective decoded signal 103 a, 103 b, . . . ,103 n to collect and provide spatial and temporal data 121 a, 121 b, . .. , 121 n. The final view processor 124 combines and uses this data 121a, 121 b, . . . , 121 n to produce the single view signal 107. Thissingle view signal 107 can be sent to a display 202 for immediate videoplayback, stored in memory 204 for later use, encoded by another encoder206 prior to transmission elsewhere for other uses, or passed ontoanother application for further processing.

Referring the FIG. 6, use of temporal and spatial image information forprocessing the single view signal 107 can be visualized as shown. Forexample, as discussed above, the base view 104 a includes a sequence oftenporally adjacent video frames 105 a, including intermediate frames105 aa, 105 ab, 105 ac. Similarly, spatially adjacent dependent views104 b, 104 c include respective sequences of tenporally adjacent videoframes 104 b, 104 c, including intermediate frames 105 ba, 105 bb, 105bc, 105 ca, 105 cb, 105 cc. As depicted here, such dependent views, 104b, 104 c have video frames 105 b, 105 c, that are spatially adjacent tocorresponding video frames 105 a of the base view 104 a. For example,the central intermediate frame 105 ab of the base view 104 a correspondsto spatially adjacent central intermediate frames 105 bb, 105 cb of itsspatially adjacent dependent views 104 b, 104 c. Accordingly, inaccordance with an exemplary embodiment, when the final view processor124 (FIG. 5) is processing this central intermediate video frame 105 ab,it can use image information from the corresponding frames 105 bb, 105cb of the adjacent dependent views 104 b, 104 c. Additionally, suchprocessor 120 a can also use image information from the tenporallyadjacent video frames 105 ba, 105 bc, 105 ca, 105 cc of the dependentviews 104 b, 104 c. Further, such processor 120 a can also use imageinformation from tenporally adjacent video frames 105 aa, 105 ac withinthe base view itself, e.g., as part of an interpolation process.

Referring to FIG. 7, an exemplary embodiment of a method 300 ofproviding a single view video signal based on a MVC signal stream beginsby decoding 302 a MVC signal stream representing a plurality ofspatially related views of a scene, including a base view and at leastone dependent view, to provide a plurality of decoded video signalsrepresenting the plurality of spatially related views. Following this isprocessing 304 the plurality of decoded video signals using imageinformation from more than one of the plurality of decoded video signalsto provide a processed video signal representing one of the plurality ofspatially related views.

As discussed hereinabove, the processors 120 a, 120 b, . . . , 120 n(FIG. 5) can be implemented using various combinations of hardware(e.g., one or more microprocessors or other suitable processors),firmware and software. Such hardware portions are generally designed byintegrated circuit design systems (e.g., work stations with digitalprocessors) that create integrated circuits based on executableinstructions stored on a computer readable medium including memory(e.g., CDROM, RAM, other forms of ROM, hard drives, distributed memory,or any other suitable computer readable medium). The instructions may berepresented by any suitable language such as but not limited to hardwaredescriptor language (HDL) or other suitable language. The computerreadable medium contains the executable instructions that when executedby the integrated circuit design system causes the integrated circuitdesign system to produce an integrated circuit that includes the devicesor circuitry as set forth above. The code is executed by one or moreprocessing devices in a work station or system (not shown). As such, thedevices or circuits described herein may also be produced as integratedcircuits by such integrated circuit design systems executing suchinstructions.

Although the invention has been described in connection with specificpreferred embodiments, it should be understood that the invention asclaimed should not be unduly limited to such specific embodiments asother modifications and alternations in the method and structure of thisinvention will be apparent to those skilled in the art without departingfrom the scope and the spirit of the invention. It is intended that thefollowing claims define the scope of the present invention and thatmethods and structures within the scope of these claims and theirequivalents be covered thereby.

What is claimed is:
 1. A method, carried out by a video system forproviding a single view video signal based on a multiview video coding(MVC) signal stream, comprising: processing a MVC signal stream usingimage information from more than one of a plurality of decoded videosignals that represent a plurality of spatially related views; andproviding a processed video signal representing one of said plurality ofspatially related views based on said processing said MVC signal streamusing said image information from said more than one of said pluralityof decoded video signals that represent said plurality of spatiallyrelated views.
 2. The method of claim 1, wherein: said MVC signal streamincludes a bitstream corresponding to a base view and at least anotherbitstream corresponding to at least one dependent view; and saidprocessing a MVC signal stream comprises decoding said bitstreamcorresponding to said base view to provide one of said plurality ofdecoded video signals, and decoding said bitstream corresponding to saidat least one dependent view to provide at least another one of saidplurality of decoded video signals.
 3. The method of claim 1, wherein:said MVC signal stream includes a bitstream of which a first portioncorresponds to a base view and of which at least a second portioncorresponds to at least one dependent view; and said processing a MVCsignal stream comprises decoding said first portion of said bitstream toprovide one of said plurality of decoded video signals, and decodingsaid at least a second portion of said bitstream to provide at leastanother one of said plurality of decoded video signals.
 4. The method ofclaim 1, wherein: respective portions of said MVC signal stream includerespective temporal image information corresponding to one of aplurality of temporally adjacent video frames; respective portions ofsaid MVC signal stream include respective spatial image informationcorresponding to one of a plurality of sequences of spatially adjacentvideo frames; and said processing said MVC signal stream comprisesprocessing said plurality of decoded video signals using at least one ofsaid temporal and spatial image information.
 5. The method of claim 4,wherein said processing said MVC signal stream comprises: processing atleast one of said plurality of decoded video signals representing atleast one dependent view to provide said at least one of said temporaland spatial image information related to said at least one dependentview; and processing another one of said plurality of decoded videosignals representing a base view using said at least one of saidtemporal and spatial image information related to said at least onedependent view to provide said processed video signal representing saidbase view.
 6. The method of claim 1, wherein said processing said MVCsignal stream comprises: processing at least one of said plurality ofdecoded video signals representing at least one dependent view toprovide image information related to said at least one dependent view;and processing another one of said plurality of decoded video signalsrepresenting a base view using said image information related to said atleast one dependent view to provide said processed video signalrepresenting said base view.
 7. An apparatus including a video systemfor providing a single view video signal based on a multiview videocoding (MVC) signal stream, comprising: a decoder adapted to decode aMVC signal stream representing a plurality of spatially related views ofa scene, including a base view and at least one dependent view, toprovide a plurality of decoded video signals representing said pluralityof spatially related views; and one or more video processors adapted toprocess said plurality of decoded video signals using image informationfrom more than one of said plurality of decoded video signals to providea processed video signal representing one of said plurality of spatiallyrelated views using said image information from said more than one ofsaid plurality of decoded video signals that represent said plurality ofspatially related views.
 8. The apparatus of claim 7, wherein: said MVCsignal stream includes a bitstream corresponding to said base view andat least another bitstream corresponding to said at least one dependentview; and said decoder is adapted to decode said bitstream correspondingto said base view to provide one of said plurality of decoded videosignals, and decode said bitstream corresponding to said at least onedependent view to provide at least another one of said plurality ofdecoded video signals.
 9. The apparatus of claim 7, wherein: said MVCsignal stream includes a bitstream of which a first portion correspondsto said base view and of which at least a second portion corresponds tosaid at least one dependent view; and said decoder is adapted to decodesaid first portion of said bitstream to provide one of said plurality ofdecoded video signals, and decode said at least a second portion of saidbitstream to provide at least another one of said plurality of decodedvideo signals.
 10. The apparatus of claim 7, wherein: respectiveportions of said MVC signal stream include respective temporal imageinformation corresponding to one of a plurality of temporally adjacentvideo frames; respective portions of said MVC signal stream includerespective spatial image information corresponding to one of a pluralityof sequences of spatially adjacent video frames; and said one or morevideo processors are adapted to process said plurality of decoded videosignals using at least one of said temporal and spatial imageinformation.
 11. The apparatus of claim 10, wherein said one or morevideo processors are adapted to: process at least one of said pluralityof decoded video signals representing said at least one dependent viewto provide said at least one of said temporal and spatial imageinformation related to said at least one dependent view; and processanother one of said plurality of decoded video signals representing saidbase view using said at least one of said temporal and spatial imageinformation related to said at least one dependent view to provide saidprocessed video signal representing said base view.
 12. The apparatus ofclaim 7, wherein said one or more video processors are adapted to:process at least one of said plurality of decoded video signalsrepresenting said at least one dependent view to provide imageinformation related to said at least one dependent view; and processanother one of said plurality of decoded video signals representing saidbase view using said image information related to said at least onedependent view to provide said processed video signal representing saidbase view.
 13. A non-transitory computer readable medium comprising aplurality of executable instructions that, when executed by anintegrated circuit design system, cause the integrated circuit designsystem to produce at least one integrated circuit (IC) for providing asingle view video signal based on a multiview video coding (MVC) signalstream and including: a decoder adapted to decode a MVC signal streamrepresenting a plurality of spatially related views of a scene,including a base view and at least one dependent view, to provide aplurality of decoded video signals representing said plurality ofspatially related views; and one or more video processors adapted toprocess said plurality of decoded video signals using image informationfrom more than one of said plurality of decoded video signals to providea processed video signal representing one of said plurality of spatiallyrelated views using said image information from said more than one ofsaid plurality of decoded video signals that represent said plurality ofspatially related views.
 14. The non-transitory computer readable mediumof claim 13, wherein: said MVC signal stream includes a bitstreamcorresponding to said base view and at least another bitstreamcorresponding to said at least one dependent view; and said decoder isadapted to decode said bitstream corresponding to said base view toprovide one of said plurality of decoded video signals, and decode saidbitstream corresponding to said at least one dependent view to provideat least another one of said plurality of decoded video signals.
 15. Thenon-transitory computer readable medium of claim 13, wherein: said MVCsignal stream includes a bitstream of which a first portion correspondsto said base view and of which at least a second portion corresponds tosaid at least one dependent view; and said decoder is adapted to decodesaid first portion of said bitstream to provide one of said plurality ofdecoded video signals, and decode said at least a second portion of saidbitstream to provide at least another one of said plurality of decodedvideo signals.
 16. The non-transitory computer readable medium of claim13, wherein: respective portions of said MVC signal stream includerespective temporal image information corresponding to one of aplurality of temporally adjacent video frames; respective portions ofsaid MVC signal stream include respective spatial image informationcorresponding to one of a plurality of sequences of spatially adjacentvideo frames; and said one or more video processors are adapted toprocess said plurality of decoded video signals using at least one ofsaid temporal and spatial image information.
 17. The non-transitorycomputer readable medium of claim 16, wherein said one or more videoprocessors are adapted to: process at least one of said plurality ofdecoded video signals representing said at least one dependent view toprovide said at least one of said temporal and spatial image informationrelated to said at least one dependent view; and process another one ofsaid plurality of decoded video signals representing said base viewusing said at least one of said temporal and spatial image informationrelated to said at least one dependent view to provide said processedvideo signal representing said base view.
 18. The non-transitorycomputer readable medium of claim 13, wherein said one or more videoprocessors are adapted to: process at least one of said plurality ofdecoded video signals representing said at least one dependent view toprovide image information related to said at least one dependent view;and process another one of said plurality of decoded video signalsrepresenting said base view using said image information related to saidat least one dependent view to provide said processed video signalrepresenting said base view.